#ceph IRC Log


IRC Log for 2012-11-15

Timestamps are in GMT/BST.

[0:00] <jlogan1> Here is the debug output
[0:00] <jlogan1> 2012-11-14 22:58:51.349+0000: 9494: debug : virStorageBackendRBDCreateVol:418 : Creating RBD image data/test7-sef.prod.msrch-disk1 with size 20971520
[0:00] <jlogan1> 2012-11-14 22:58:51.349+0000: 9494: debug : virStorageBackendRBDOpenRADOSConn:62 : Found Cephx username: (null)
[0:00] <jlogan1> 2012-11-14 22:58:51.349+0000: 9494: debug : virStorageBackendRBDOpenRADOSConn:120 : Not using cephx authorization
[0:00] <jlogan1> 2012-11-14 22:58:51.350+0000: 9494: debug : virStorageBackendRBDOpenRADOSConn:135 : Found 3 RADOS cluster monitors in the pool configuration
[0:00] <jlogan1> 2012-11-14 22:58:51.350+0000: 9494: debug : virStorageBackendRBDOpenRADOSConn:159 : RADOS mon_host has been set to:,,,
[0:00] <jlogan1> 2012-11-14 22:58:51.354+0000: 9494: debug : virSecretFree:15438 : secret=(nil)
[0:00] <jlogan1> 2012-11-14 22:58:51.354+0000: 9494: error : virSecretFree:15443 : Invalid secret: virSecretFree
[0:00] <jlogan1> 2012-11-14 22:58:51.356+0000: 9494: error : virStorageBackendRBDCreateVol:442 : internal error failed to create volume 'data/test7-sef.prod.msrch-disk1'
[0:00] <jlogan1> 2012-11-14 22:58:51.357+0000: 9494: debug : virStorageBackendRBDCloseRADOSConn:191 : Closing RADOS IoCTX
[0:00] <jlogan1> 2012-11-14 22:58:51.357+0000: 9494: debug : virStorageBackendRBDCloseRADOSConn:198 : Closing RADOS connection
[0:00] <jlogan1> 2012-11-14 22:58:51.358+0000: 9494: debug : virStorageBackendRBDCloseRADOSConn:205 : RADOS connection existed for 0 seconds
[0:00] <jlogan1> 2012-11-14 22:58:51.358+0000: 9494: debug : virStoragePoolFree:12592 : pool=0x7ffbf00027c0
[0:01] <joshd> jlogan1: ok, that would be a bug in the storage pool driver when cephx is disabled, virSecretFree does not accept NULL like other cleanup actions
[0:01] <jlogan1> joshd: I think authentication is OK. See the output from vol-list:
[0:01] <jlogan1> root@ceph01-sef:/etc/libvirt/storage# virsh vol-list data
[0:01] <jlogan1> Name Path
[0:01] <jlogan1> -----------------------------------------
[0:01] <jlogan1> foo data/foo
[0:01] <jlogan1> test1-sef.prod.msrch-disk1 data/test1-sef.prod.msrch-disk1
[0:01] <jlogan1> test6-sef.prod.msrch-disk1 data/test6-sef.prod.msrch-disk1
[0:01] <Psi-jack> Anyone heard of pastebin.ca? ;)
[0:02] <jlogan1> Psi-jack: it was border line... ;)
[0:02] <Psi-jack> 12+ lines is "border line?"
[0:03] * dmick (~dmick@2607:f298:a:607:5144:73e6:7ad0:5110) has joined #ceph
[0:04] * LarsFronius (~LarsFroni@2a02:8108:380:12:dcb9:8663:3b6f:d0c7) Quit (Quit: LarsFronius)
[0:05] <jlogan1> joshd: I'll look at cephx and get that setup, any pointers?
[0:07] <joshd> jlogan1: http://eu.ceph.com/docs/master/cluster-ops/authentication/
[0:07] * iconz55 (~Brandon@host-173-246-63-229.biznesshosting.net) Quit (Quit: Leaving.)
[0:08] * davidz (~Adium@ has joined #ceph
[0:10] * timmclaughlin (~timmclaug@ Quit (Read error: Operation timed out)
[0:22] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) has left #ceph
[0:22] * wonko_be_ (bernard@november.openminds.be) has joined #ceph
[0:23] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Quit: Leaving.)
[0:24] * wonko_be (bernard@november.openminds.be) Quit (Read error: Connection reset by peer)
[0:32] <jlogan1> The doc steps are not working on my cluster.
[0:32] <jlogan1> http://pastebin.ca/2251633
[0:33] <dmick> jlogan1: did you already have a client.admin key in that keyring file?
[0:34] <jlogan1> Yes
[0:34] <jlogan1> but I tried to remove that, but did not help
[0:34] <jlogan1> I did save a copy
[0:34] <Psi-jack> Holy crap. LOl
[0:35] <Psi-jack> I've never had a VM issue like this, where just booting up, everything suddenly runs out of memory, DURING init, that it OOM kills everything until nothing is left to kill.
[0:36] <jlogan1> there is nothing interesting on my cluster if I'm better off making a new cephfs.
[0:39] <Psi-jack> Sheash..
[0:40] * davidz (~Adium@ Quit (Quit: Leaving.)
[0:40] <dmick> jlogan1: I'm thinking you want to elide mds there but I'm not sure, checking
[0:40] <dmick> the DOC URLs just changed :)
[0:42] * yoshi (~yoshi@p8092-ipngn502marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[0:43] <dmick> ah: jlogan1: slight typo
[0:43] <dmick> no '*' on the mds 'allow'
[0:44] <dmick> (the docs are correct AFAICT)
[0:45] <jlogan1> what link are you using now?
[0:45] <dmick> http://ceph.com/docs/master/rados/operations/authentication/#the-client-admin-key
[0:46] <dmick> (eu. works too)
[0:46] <dmick> the docs are autobuilt on push, and the push just happened
[0:46] <dmick> some reorganization
[0:48] * tnt (~tnt@18.68-67-87.adsl-dyn.isp.belgacom.be) Quit (Read error: Operation timed out)
[0:55] <jlogan1> I added the cephx lines to ceph.conf and I ran a new mkceph.
[0:57] <jlogan1> the mon keys were made by default: /var/lib/ceph/mon/ceph-*/keyring
[0:59] * miroslav (~miroslav@ has joined #ceph
[1:01] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has left #ceph
[1:02] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[1:02] * ChanServ sets mode +o scuttlemonkey
[1:03] <jlogan1> Now that this is in place how do I make a key for rbd pools?
[1:04] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[1:11] * fmarchand2 (~fmarchand@ Quit (Quit: Yaaic - Yet another Android IRC client - http://www.yaaic.org)
[1:14] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has joined #ceph
[1:15] <joshd> jlogan1: you can use client.admin's key, or create a new client with ceph auth get-or-create
[1:17] <sagewk> slang: there?
[1:21] <jlogan1> I'm using some hints from: http://ceph.com/docs/master/rbd/rbd-openstack/
[1:22] <joshd> jlogan1: which ceph version are you using?
[1:25] <jlogan1> newest testing version. 0.54
[1:25] <jlogan1> with libvirt 1.0
[1:39] <jlogan1> I have a new error:
[1:39] <jlogan1> virStorageBackendRBDOpenRADOSConn:171 : internal error failed to connect to the RADOS monitor on:,,,
[1:39] <jlogan1> I verified via telnet the port is listening
[1:39] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[1:41] <joshd> try connecting with the cli as the same user (i.e. rbd ls --id foo --keyring /path/to/keyring) where client.foo is your rados user
[1:43] * maxiz (~pfliu@ Quit (Quit: Ex-Chat)
[1:45] <jlogan1> joshd: http://pastebin.ca/2251650
[1:45] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[1:46] <jlogan1> joshd: updated http://pastebin.ca/2251651
[1:47] <joshd> jlogan1: /etc/ceph/ceph.keyring contains the key for client.data?
[1:48] <jlogan1> joshd: /etc/ceph/ceph.keyring is the main keyring. client.data is in there.
[1:49] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[1:49] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[1:51] <joshd> jlogan1: double check the key in the keyring vs 'ceph auth list'? since it's not even able to authenticate to the monitors, the key must not match
[1:53] * sjustlaptop1 (~sam@m920436d0.tmodns.net) has joined #ceph
[1:54] <jlogan1> joshd: when I made client.data.keyring then rbd ls -p data --id data --keyring /etc/ceph/client.data.keyring started to work.
[1:55] <jlogan1> but I'm not sure now how to make my secret.xml and pool.xml file for libvirt work any better...
[1:56] <jlogan1> some progress... I changed client.data to data in : <auth username='data' type='ceph'>
[1:56] <jlogan1> root@ceph01-sef:/etc/ceph# virsh vol-create-as data test7-sef.prod.msrch-disk1 20971520
[1:56] <jlogan1> Vol test7-sef.prod.msrch-disk1 created
[1:56] * vjarjadian (~IceChat7@5ad6d001.bb.sky.com) Quit (Ping timeout: 480 seconds)
[1:56] <lurbs> jlogan1: I had the same issue, was about to say.
[1:57] <lurbs> It's not immediately obvious that the username needs s/client.//
[1:57] <jlogan1> I can use virsh to make a volume from the pool. Let me try foreman.
[1:58] <lurbs> libvirt started supporting RBD pools in 0.9.14 ish?
[1:58] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[1:58] <lurbs> Pity the Ubuntu cloud archive only has 0.9.13. :-/
[2:01] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[2:01] <jlogan1> I'm not sure the version. I'm using libvirt 1.0 from source.
[2:03] <jlogan1> Looks like cgroup may hav having an issue.
[2:03] <jlogan1> 2012-11-15 00:58:57.763+0000: 17580: error : virFileOpenAs:1077 : failed to open file 'data/test7-sef.prod.msrch-disk1': No such file or directory
[2:03] <jlogan1> 2012-11-15 00:58:57.763+0000: 17580: error : virStorageFileGetMetadataRecurse:962 : cannot open file 'data/test7-sef.prod.msrch-disk1': No such file or directory
[2:07] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:08] <jlogan1> lurbs: Do you have libvirt working with pools today?
[2:09] <lurbs> No, I'm still using 0.9.13.
[2:10] <lurbs> Which seems to work fine with manually created RBD disk definitions, but it doesn't understand the RBD pool stuff.
[2:11] <joshd> jlogan1: is that when foreman's trying to use it?
[2:12] <joshd> jlogan1: libvirt doesn't have an api to tell an application how to attach a disk from a given pool, so if foreman doesn't know about rbd, it'll try to use it like a file
[2:12] <jlogan1> Yes. libvirt posted that error when foreman tried to create the vm.
[2:13] <joshd> jlogan1: you can look at the xml for the domain it's trying to create to be sure
[2:13] <jlogan1> I was worried about that.
[2:13] <jlogan1> I don't know if I can see the xml it created for the domain.
[2:13] <jlogan1> I'll have to look tomorrow.
[2:13] <jlogan1> I got farther this time with having the rbd pool known to libvirt.
[2:14] <jlogan1> I know I can make VMs by hand, but it's so close to working with 1-click deploy in foreman...
[2:16] <joshd> if you know ruby, it's probably not hard to make foreman attach rbd disks correctly
[2:17] <jlogan1> a team mate knows ruby, I know a tiny bit.
[2:17] * sagelap (~sage@2607:f298:a:607:d13b:da08:ecfe:a0e4) Quit (Ping timeout: 480 seconds)
[2:17] <jlogan1> so you are thinking if we modify the template to use the volume just created it may work?
[2:18] * sagelap (~sage@200.sub-70-197-130.myvzw.com) has joined #ceph
[2:18] <joshd> possibly, I don't know what else foreman tries to do to it
[2:19] <jlogan1> it uses fog (I think), so sometimes it's a bit hidden.
[2:19] <jlogan1> I'll work on getting the xml it creates tomorrow which should give us some ideas
[2:24] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[2:24] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[2:24] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[2:25] * sjusthm (~sam@m920436d0.tmodns.net) has joined #ceph
[2:26] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Remote host closed the connection)
[2:27] * sjustlaptop1 (~sam@m920436d0.tmodns.net) Quit (Read error: Connection reset by peer)
[2:28] <jlogan1> good call, the template is off.
[2:28] <jlogan1> <disk type='file' device='disk'>
[2:28] <jlogan1> <driver name='qemu' type='raw'/>
[2:28] <jlogan1> <source file='data/test2-sef.prod.msrch-disk1'/>
[2:28] <jlogan1> <target dev='vda' bus='virtio'/>
[2:28] <jlogan1> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
[2:28] <jlogan1> </disk>
[2:29] <jlogan1> and since libvirt will only be on ceph we can just change it here...
[2:29] <jlogan1> thanks for the help today
[2:29] <joshd> no problem
[2:33] * adjohn (~adjohn@ Quit (Quit: adjohn)
[2:34] <lurbs> Looks like RBD pool support was actually added in libvirt-bin 0.9.13, but the Ubuntu packages don't enable it.
[2:34] * sjustlaptop (~sam@m920436d0.tmodns.net) has joined #ceph
[2:34] * LarsFronius (~LarsFroni@95-91-242-159-dynip.superkabel.de) has joined #ceph
[2:35] * LarsFronius (~LarsFroni@95-91-242-159-dynip.superkabel.de) Quit ()
[2:37] * jlogan1 (~Thunderbi@2600:c00:3010:1:fc51:3370:98cf:1d11) Quit (Ping timeout: 480 seconds)
[2:37] <joshd> lurbs: they're enabling it for 13.04
[2:38] * lurbs boots up his 13.04 VM and checks if it's there already. :)
[2:39] <lurbs> Nope.
[2:39] * sjusthm (~sam@m920436d0.tmodns.net) Quit (Quit: Leaving.)
[2:51] * buck (~buck@bender.soe.ucsc.edu) has left #ceph
[2:53] * maxiz (~pfliu@ has joined #ceph
[3:06] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[3:07] * sagelap (~sage@200.sub-70-197-130.myvzw.com) Quit (Ping timeout: 480 seconds)
[3:08] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[3:08] * gregaf (~Adium@2607:f298:a:607:b01b:ef4c:9fb6:39c9) has joined #ceph
[3:08] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit ()
[3:22] * dmick (~dmick@2607:f298:a:607:5144:73e6:7ad0:5110) Quit (Quit: Leaving.)
[3:33] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[3:39] * sagelap (~sage@ has joined #ceph
[3:56] <elder> I'm back.
[3:59] * mdrnstm (~mdrnstm@206-169-78-213.static.twtelecom.net) Quit (Quit: Leaving.)
[4:17] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[4:18] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[4:18] * plut0 (~cory@pool-96-236-43-69.albyny.fios.verizon.net) has left #ceph
[4:25] * chutzpah (~chutz@ Quit (Quit: Leaving)
[4:26] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[4:34] * jlogan1 (~Thunderbi@2600:c00:3010:1:fc51:3370:98cf:1d11) has joined #ceph
[4:38] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Remote host closed the connection)
[4:39] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[4:40] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Remote host closed the connection)
[4:41] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[4:48] * sjustlaptop (~sam@m920436d0.tmodns.net) Quit (Quit: Leaving.)
[4:56] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[4:59] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[4:59] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[5:23] * yoshi (~yoshi@p8092-ipngn502marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[5:23] * houkouonchi-work (~linux@ Quit (Read error: Connection reset by peer)
[5:29] * houkouonchi-work (~linux@ has joined #ceph
[5:57] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[6:06] * yoshi (~yoshi@p8092-ipngn502marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[6:09] <Psi-jack> Well, I am officially utilizing ceph as of now, finally. :d
[6:22] <Psi-jack> Using ceph as an rbd device, does that basically pretty much thin provision the storage for each rbd? It looks like it does, because a 20GB disk I added to a VM is only using up 4GB actual space, so far.
[6:27] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[6:27] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[6:29] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[6:29] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[6:49] * jlogan1 (~Thunderbi@2600:c00:3010:1:fc51:3370:98cf:1d11) Quit (Ping timeout: 480 seconds)
[7:00] * rweeks (~rweeks@ has joined #ceph
[7:04] * jlogan1 (~Thunderbi@ has joined #ceph
[7:07] * tnt (~tnt@140.20-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[7:09] * tryggvil_ (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[7:10] * grifferz (~andy@specialbrew.392abl.bitfolk.com) has left #ceph
[7:11] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Ping timeout: 480 seconds)
[7:11] * tryggvil_ is now known as tryggvil
[7:28] * jabadia (~jabadia@ has joined #ceph
[7:31] <jabadia> rpm -i ceph-radosgw-0.53-6.el6.x86_64.rpm
[7:31] <jabadia> error: Failed dependencies:
[7:31] <jabadia> libfcgi.so.0()(64bit) is needed by ceph-radosgw-0.53-6.el6.x86_64
[7:31] <jabadia> mod_fcgid is needed by ceph-radosgw-0.53-6.el6.x86_64
[7:31] <jabadia> [root@rados-gw ~]#
[7:31] <jabadia> any idea?
[7:31] <rweeks> you need the fast CGI library for the rados gateway to work
[7:32] <jabadia> ok, thanks, I look for those RPM's in Redhat
[7:40] <tontsa> jabadia, you need to enable epel-repo to get fcgi
[7:41] <jabadia> Thanks!
[7:45] * rweeks (~rweeks@ Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[7:50] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Quit: Leaving.)
[8:00] * raso (~raso@deb-multimedia.org) Quit (Quit: WeeChat 0.3.8)
[8:01] * raso (~raso@deb-multimedia.org) has joined #ceph
[8:04] * loicd (~loic@magenta.dachary.org) has joined #ceph
[8:07] * yehuda_hm (~yehuda@2602:306:330b:a40:8d7b:6204:315:4be) Quit (Remote host closed the connection)
[8:14] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[8:20] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[8:50] * Kioob`Taff1 (~plug-oliv@local.plusdinfo.com) has joined #ceph
[8:56] * houkouonchi-work (~linux@ Quit (Quit: Client exiting)
[9:04] * ScOut3R_ (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) has joined #ceph
[9:06] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[9:20] * tnt (~tnt@140.20-67-87.adsl-dyn.isp.belgacom.be) Quit (Read error: Operation timed out)
[9:35] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:36] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:39] * loicd1 (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[9:41] * MikeMcClurg (~mike@ has joined #ceph
[9:42] * loicd1 (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Read error: Connection reset by peer)
[9:42] * loicd2 (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[9:46] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Ping timeout: 480 seconds)
[9:52] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) has joined #ceph
[10:08] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[10:09] * Leseb (~Leseb@ has joined #ceph
[10:27] * wilson (~wilson@CPE001c1025d510-CM001ac317ccea.cpe.net.cable.rogers.com) Quit (Read error: Connection reset by peer)
[10:27] * wilson (~wilson@CPE001c1025d510-CM001ac317ccea.cpe.net.cable.rogers.com) has joined #ceph
[10:34] * BManojlovic (~steki@ has joined #ceph
[10:41] * miroslav (~miroslav@ Quit (Quit: Leaving.)
[10:45] * maxiz (~pfliu@ Quit (Quit: Ex-Chat)
[10:47] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Quit: Leaving.)
[10:55] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[11:04] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[11:05] * jlogan1 (~Thunderbi@ Quit (Ping timeout: 480 seconds)
[11:08] <masterpe> Ok, all my OSD are full
[11:09] <masterpe> (95%
[11:09] <masterpe> google says taht I can do: ceph mon tell \* injectargs '--mon_osd_full_ratio 0.98'
[11:09] <masterpe> to give me write access
[11:10] <masterpe> so that i can remove the big object
[11:10] <masterpe> But it doesn't work
[11:12] <masterpe> f
[11:18] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[11:18] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:21] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[11:21] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:23] * epitron (~epitron@bito.ponzo.net) Quit (Remote host closed the connection)
[11:24] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[11:27] * BManojlovic (~steki@ has joined #ceph
[11:28] * epitron (~epitron@bito.ponzo.net) has joined #ceph
[11:28] * epitron (~epitron@bito.ponzo.net) Quit (Remote host closed the connection)
[11:28] * epitron (~epitron@bito.ponzo.net) has joined #ceph
[11:29] * epitron (~epitron@bito.ponzo.net) Quit (Remote host closed the connection)
[11:30] * yoshi (~yoshi@p8092-ipngn502marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[11:31] * epitron (~epitron@bito.ponzo.net) has joined #ceph
[12:00] * jlogan (~Thunderbi@2600:c00:3010:1:2c13:903:77f6:157d) has joined #ceph
[12:04] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) Quit (Ping timeout: 480 seconds)
[12:23] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) Quit (Remote host closed the connection)
[12:23] * jlogan (~Thunderbi@2600:c00:3010:1:2c13:903:77f6:157d) Quit (Quit: jlogan)
[12:24] * silversurfer (~silversur@124x35x68x250.ap124.ftth.ucom.ne.jp) has joined #ceph
[12:56] * Kioob`Taff1 (~plug-oliv@local.plusdinfo.com) Quit (Quit: Leaving.)
[13:00] * ramsay_za (~ramsay_za@ Quit (Remote host closed the connection)
[13:00] * verwilst (~verwilst@d5152FEFB.static.telenet.be) has joined #ceph
[13:25] * CristianDM (~CristianD@host192.186-124-184.telecom.net.ar) has joined #ceph
[13:25] <CristianDM> Hi.
[13:25] <CristianDM> I am ceph 0.48.2 inside proxmox and 0.50 currently for osd, and mons
[13:26] <CristianDM> If I upgrade osd and mons to 0.54, the ceph 0.48.2 can comunicate normally?
[13:27] * jks (~jks@3e6b7199.rev.stofanet.dk) Quit (Ping timeout: 480 seconds)
[13:27] <Psi-jack> Why are you mixing that all up? ;)
[13:28] <CristianDM> proxmox can´t enable upgrade to development versions
[13:28] <CristianDM> So I can´t upgrade inside proxmox
[13:29] <Psi-jack> As the Proxmox documentation also says, you shouldn't be running ceph ON the proxmox hosts themselves. You should only be running your Ceph storage cluster on separate storage systems.
[13:29] <Psi-jack> Generally, in surplus of 3.
[13:29] <CristianDM> Yes, I have ceph storage in other hosts
[13:30] <CristianDM> but proxmox have ceph for rbd
[13:30] <Psi-jack> Ahhhh, I see what your asking.
[13:30] <Psi-jack> So far, I dunno. I'm using ceph's official repos.
[13:30] * KindOne (KindOne@h210.25.131.174.dynamic.ip.windstream.net) Quit (Ping timeout: 480 seconds)
[13:30] <Psi-jack> Which only has 0.48
[13:31] <Psi-jack> Which is basically exactly what proxmox's repos have as well.
[13:31] * loicd2 (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Ping timeout: 480 seconds)
[13:31] <CristianDM> Proxmox get the packages and put them inside the proxmox repo
[13:32] <tnt> RBD should stay compatible AFAIK.
[13:32] <tnt> (but I'm no dev ...)
[13:32] <Psi-jack> Yeah, I would think to, tnt.
[13:32] <tnt> I have a 0.48.2 cluster and use the rbd client from stock kernel and it works fine.
[13:33] <Psi-jack> Heh
[13:33] <Psi-jack> I've been testing out ceph with a new testing VM installation.
[13:33] <Psi-jack> Frustrated, a little, that ceph doesn't work so well on ZFS, but it works "ok" on an ext4 subvolume, just a little slower.
[13:34] <Psi-jack> Well, actually, horrendously slow, so far. LOL
[13:36] * KindOne (KindOne@h210.25.131.174.dynamic.ip.windstream.net) has joined #ceph
[13:38] * laevar (~jochen@laevar.de) Quit (Quit: leaving)
[13:40] * CristianDM (~CristianD@host192.186-124-184.telecom.net.ar) Quit (Ping timeout: 480 seconds)
[13:40] * a (~CristianD@host192.186-124-184.telecom.net.ar) has joined #ceph
[13:40] * a is now known as CristianDM
[13:41] <CristianDM> a
[13:41] <CristianDM> joshd: You know if works with rbd client 0.48.2 and osd/mon servers with 0.54?
[13:45] * jochen (~jochen@laevar.de) has joined #ceph
[13:49] <CristianDM> Any kwnow if cephfs works with cachefilesd?
[13:50] <CristianDM> FS-Cache
[13:54] * Robe (robe@amd.co.at) Quit (Quit: server maintenance!)
[14:01] * KindOne (KindOne@h210.25.131.174.dynamic.ip.windstream.net) Quit (Remote host closed the connection)
[14:11] * KindOne (KindOne@h210.25.131.174.dynamic.ip.windstream.net) has joined #ceph
[14:39] * jks (~jks@3e6b7199.rev.stofanet.dk) has joined #ceph
[14:42] * Kioob`Taff1 (~plug-oliv@local.plusdinfo.com) has joined #ceph
[14:45] * timmclaughlin (~timmclaug@ has joined #ceph
[14:49] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) has joined #ceph
[14:51] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[14:52] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Read error: Operation timed out)
[14:53] * LarsFronius_ is now known as LarsFronius
[15:03] <Psi-jack> Heh, well, so far, ceph as an rbd provider for Proxmox VE to a VM guest, is a bit slow, at least to install the base OS onto. Afterwards, it's still a little slow, but not quite as much so.
[15:03] <tnt> What RBD client is it using ?
[15:03] <Psi-jack> ceph's rbd client.
[15:03] <tnt> thre is like 3 ...
[15:04] <tnt> kernel, fuse, qemu
[15:04] <Psi-jack> It's either qemu or kernel, then.
[15:05] <CristianDM> I use proxmox with ceph
[15:05] <Psi-jack> It took a few hours to install Ubuntu 12.04 Server base install, where-as over NFSv4->qcow2 disks, it takes about 20 minutes.
[15:05] <CristianDM> And share inside a VM with NFS
[15:05] <CristianDM> This is slow for small files
[15:06] <Psi-jack> I'm using ceph for the OS disk itself, and NFSv4 for files, such as NFS for the webserver content, which actually ends up being really fast.
[15:06] <tnt> Here I install on local images then upload the entire image at once to RBD, much faster. Then most VM access are caches anyway and not much writes ...
[15:06] * gucki_ (~smuxi@HSI-KBW-082-212-034-021.hsi.kabelbw.de) has joined #ceph
[15:06] <Psi-jack> Most of my VM's now, use NFSv4 to access their qcow2 disks, then for additional storage, NFSv4 within the guest.
[15:06] * gucki_ (~smuxi@HSI-KBW-082-212-034-021.hsi.kabelbw.de) Quit (Remote host closed the connection)
[15:07] <Psi-jack> tnt: Yeah, that's why I'm testing out ceph. I want to see how it performs versus NFSv4.
[15:07] <CristianDM> Psi-jack: But have HA storage for NFSv4?
[15:07] <Psi-jack> CristianDM: Not really, no.
[15:08] <CristianDM> I have the same issue
[15:08] <CristianDM> HA vs Speed
[15:08] <CristianDM> :P
[15:08] <tnt> Here I store most of the big content in RGW and the small files are all cached in memory.
[15:10] <CristianDM> But when works with 500GB of files of 5Kb...
[15:10] <CristianDM> Memory not is an option
[15:11] <Psi-jack> I don't care of t he Guest OS boots up slow, or is slower doing OS based tasks, like updates and such.
[15:11] <Psi-jack> My MAIN issue is the host OS on NFSv4->qcow2 causes annoyance, if the NAS server goes down.
[15:12] <tnt> CristianDM: well we store thumbnails in RGW ... they're like 10 Kb. Don't quite have 500 Gb of them though. more like a couple Go.
[15:12] <Psi-jack> When that happends, I literaly have to practically force kill all the attached VM's amongst 4 hypervisors, force umount the NFS volume, then start them back up.
[15:12] <Psi-jack> I'm trying out ceph to see if I can actually get this thing to allow higher availability so if one storage server goes down for a few minutes, it's not the end of the world
[15:24] * maxiz (~pfliu@ has joined #ceph
[15:25] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[15:25] <jefferai> If I have to reboot machines for a kernel upgrade, should it generally be safe for me to do so as long as I reboot them one by one and let the cluster synchronize in between?
[15:26] <jefferai> or should I somehow remove a machine from the cluster first, then reboot, then re-add?
[15:27] <tnt> no, should be fine.
[15:27] <tnt> You don't even have to wait for ceph to sync anything, just reboot
[15:28] <jefferai> ok, even if this is also taking down some of the monitors?
[15:28] <tnt> as long as there is enough mon left, yes
[15:28] <jefferai> Like, I have three boxes, each has OSDs and is a monitor
[15:28] <jefferai> right, so what is "enough mon"? :-)
[15:28] <tnt> reboot one by one and wait for ceph to display HEALTH_OK after the reboot of each.
[15:28] <tnt> 2
[15:28] <jefferai> tnt: ok, that's what I figured
[15:28] <jefferai> thanks
[15:31] * nhorman (~nhorman@nat-pool-rdu.redhat.com) has joined #ceph
[15:34] <Psi-jack> hehe
[15:34] <Psi-jack> Yeah, good, I was at least right about that. :)
[15:47] <jefferai> doh
[15:48] <jefferai> they seem to have rearranged documentation
[15:48] <jefferai> now google links are invalid
[15:48] <jefferai> elder: usually a good idea for the old pages to return a permanent redirect...Google takes this into account
[15:52] * scalability-junk (~stp@188-193-211-236-dynip.superkabel.de) has joined #ceph
[15:55] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Remote host closed the connection)
[15:59] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[16:00] <jefferai> tnt: is there a specific problem with taking down the master?
[16:01] <jefferai> because I have five mons, and now rebooted one of the storage boxes (which was also mon a / 0
[16:01] <tnt> there shouldn't be any issue no.
[16:01] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) has joined #ceph
[16:01] <jefferai> and I'm getting some issues running e.g. rados lspools
[16:01] <jefferai> saying that I don't have authentication to do so
[16:01] <elder> jefferai, sorry I'm missing context.
[16:02] <jefferai> elder: google "ceph add mon"
[16:02] <jefferai> first hit is to a page that no longer exists
[16:02] <jefferai> later hits are to pages that you don't want people looking at :-)
[16:02] <elder> Ahh.
[16:02] <elder> You better not be feeling lucky.
[16:02] <jefferai> I was suggestig that those pages that have changed URLs have permanent redirects from the old one
[16:02] <tnt> jefferai: you do have all the mon listed in your config file right ?
[16:02] <jefferai> as this will tell Google how to deal with it
[16:03] <elder> I'll create a bug suggesting that.
[16:03] <elder> Thanks.
[16:03] <tnt> jefferai: it might take a bit longer because the CLI tools probably try them in sequential order and wait for timeout, but it should work.
[16:05] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[16:06] * fc (~fc@ has joined #ceph
[16:06] <jefferai> elder: no problem
[16:06] <jefferai> tnt: yeah, I rebooted one storage box and the mons on the other two dropped out
[16:06] <jefferai> I'm guessing that it's because the version of ceph also got updated
[16:06] <jefferai> to 0.54
[16:06] <jtang> i really ought to change my google profile picture
[16:06] <elder> Thanks. http://tracker.newdream.net/issues/3496
[16:07] <jtang> someone from the penguin computing booth recognised me from my google profile pic from the ceph-devel list
[16:07] <elder> I now see that "I'm feeling lucky" isn't really an option any more with that google auto-filly thing.
[16:08] <tnt> jefferai: mmm, that's annoying
[16:13] <jefferai> huh
[16:14] <jefferai> the documentation for 0.54 is now inconsistent regarding pool mappings for authentication
[16:14] <jefferai> e.g.
[16:14] <jefferai> is it "allow rwx pool foo" or "allow rwx pool=foo"
[16:14] <jefferai> I'm guesing that in 0.54 both work
[16:28] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) has joined #ceph
[16:32] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Quit: LarsFronius)
[16:35] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[16:38] * MikeMcClurg1 (~mike@ has joined #ceph
[16:38] * MikeMcClurg (~mike@ Quit (Read error: Connection reset by peer)
[16:56] * jstrunk_ (~quassel@ has joined #ceph
[16:56] * MarkS_ (~mark@irssi.mscholten.eu) has joined #ceph
[16:58] * MikeMcClurg1 (~mike@ Quit (resistance.oftc.net charm.oftc.net)
[16:58] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * jstrunk (~quassel@ Quit (resistance.oftc.net charm.oftc.net)
[16:58] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * xiu (~xiu@ Quit (resistance.oftc.net charm.oftc.net)
[16:58] * __jt__ (~james@jamestaylor.org) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * exec (~defiler@ Quit (resistance.oftc.net charm.oftc.net)
[16:58] * MooingLemur (~troy@phx-pnap.pinchaser.com) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * MarkS (~mark@irssi.mscholten.eu) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * Anticimex (anticimex@netforce.csbnet.se) Quit (resistance.oftc.net charm.oftc.net)
[16:58] * MooingLemur (~troy@phx-pnap.pinchaser.com) has joined #ceph
[16:58] * Anticimex (anticimex@netforce.csbnet.se) has joined #ceph
[16:59] * MikeMcClurg1 (~mike@ has joined #ceph
[16:59] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[16:59] * jstrunk (~quassel@ has joined #ceph
[16:59] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) has joined #ceph
[16:59] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) has joined #ceph
[16:59] * xiu (~xiu@ has joined #ceph
[16:59] * __jt__ (~james@jamestaylor.org) has joined #ceph
[16:59] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) has joined #ceph
[16:59] * exec (~defiler@ has joined #ceph
[16:59] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) Quit (Read error: Connection reset by peer)
[16:59] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) Quit (Ping timeout: 482 seconds)
[17:00] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[17:00] * jstrunk (~quassel@ Quit (Ping timeout: 482 seconds)
[17:01] <wer> I don't know what to do about this error... FastCGI: comm with server "/var/www/radosgw.fcgi" aborted: error parsing headers: duplicate header 'Status'
[17:02] <wer> I think either that, or the fact that radosgw returns a 200 on completion of a PUT is preventing a client from thinking the PUT was successful.
[17:02] <wer> shouldn't a 201 be returned at some point?
[17:08] <tnt> Not necessarely
[17:08] <timmclaughlin> I have a bunch of stuck pgs, restarting osd's did not bring them back, anyone know where to go to fix them.
[17:09] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:09] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[17:10] * jskinner (~jskinner@ has joined #ceph
[17:10] <wer> tnt: :) I have something weird happening... and I agree with your statement. What do you make of the FastCGI error? have you see that before? The cgi will end up returning an error 500 as a result, but the puts are successful :(
[17:11] * match (~mrichar1@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[17:12] <wer> similarly s3libs returns a ERROR: XmlParseFailure but the put is successful... and the duplicate header 'Status' is logged in apache.
[17:15] * verwilst (~verwilst@d5152FEFB.static.telenet.be) Quit (Quit: Ex-Chat)
[17:18] <jskinner> Seem to be having an issue with my dev ceph deployment
[17:19] <jskinner> trying to create a volume just sits at the command line
[17:19] <jskinner> and trying to list current volumes in a pool also just sits at the command line
[17:20] <jskinner> ceph health outputs following: HEALTH_WARN 908 pgs peering; 908 pgs stuck inactive; 908 pgs stuck unclean
[17:20] <wer> jskinner: did you fill it up by chance?
[17:20] <jskinner> nope - we still have 8 TB available
[17:21] <jskinner> health HEALTH_WARN 908 pgs peering; 908 pgs stuck inactive; 908 pgs stuck unclean
[17:21] <jskinner> monmap e8: 4 mons at {a=,c=,f=,g=}, election epoch 14, quorum 0,1,2,3 a,c,f,g
[17:21] <jskinner> osdmap e1216: 12 osds: 12 up, 12 in
[17:21] <jskinner> pgmap v43232: 3512 pgs: 2604 active+clean, 908 peering; 9398 bytes data, 12607 MB used, 8004 GB / 8016 GB avail
[17:21] <jskinner> mdsmap e777: 1/1/1 up {0=a=up:replay}
[17:27] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[17:31] <wer> jskinner: isn't an even number of mons a no no? If you had none then service would certainly interrupt. Perhaps you should try 4 instead of four. Or am I reading that paste wrong?
[17:31] * ScOut3R_ (~ScOut3R@catv-80-98-44-93.catv.broadband.hu) Quit (Remote host closed the connection)
[17:31] <wer> 3 instead of 4...
[17:32] <jskinner> yeah I know what you meant
[17:32] <jskinner> yyeah
[17:32] <jskinner> we had 5
[17:32] <jskinner> actually
[17:34] <wer> maybe shoot one in the head... I only spoke up cause that is all I can think of and no one else is guessing :P
[17:35] * aliguori (~anthony@ has joined #ceph
[17:36] * loicd (~loic@207.209-31-46.rdns.acropolistelecom.net) Quit (Ping timeout: 480 seconds)
[17:39] * jlogan (~Thunderbi@2600:c00:3010:1:1ccf:467e:284:aea8) has joined #ceph
[17:40] * sagelap1 (~sage@109.sub-70-197-147.myvzw.com) has joined #ceph
[17:40] * BManojlovic (~steki@ Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:41] * sagelap (~sage@ Quit (Ping timeout: 480 seconds)
[17:42] <dspano> jskinner: I have that happen every once in a while when one of my OSDs has an out of order nic bond.
[18:08] * tnt (~tnt@212-166-48-236.win.be) Quit (Read error: Operation timed out)
[18:10] * buck (~buck@bender.soe.ucsc.edu) has joined #ceph
[18:11] * gaveen (~gaveen@ has joined #ceph
[18:14] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[18:16] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[18:18] * sagelap (~sage@2607:f298:a:607:88df:e5ed:8448:a287) has joined #ceph
[18:19] * fc (~fc@ Quit (Quit: leaving)
[18:21] * sagelap1 (~sage@109.sub-70-197-147.myvzw.com) Quit (Ping timeout: 480 seconds)
[18:21] * loicd (~loic@jem75-2-82-233-234-24.fbx.proxad.net) has joined #ceph
[18:21] * tnt (~tnt@140.20-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:22] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Quit: Leaving.)
[18:27] * Leseb (~Leseb@ Quit (Quit: Leseb)
[18:36] * sagewk (~sage@2607:f298:a:607:1d79:430b:893e:17a4) Quit (Quit: Leaving.)
[18:52] * MikeMcClurg1 (~mike@ Quit (Quit: Leaving.)
[18:53] * CristianDM (~CristianD@host192.186-124-184.telecom.net.ar) Quit ()
[18:56] <jtang> anyone tried fusion-io cards with ceph?
[18:56] * loicd (~loic@jem75-2-82-233-234-24.fbx.proxad.net) Quit (Quit: Leaving.)
[18:59] * drokita (~drokita@ has joined #ceph
[19:03] * jabadia (~jabadia@ Quit (Remote host closed the connection)
[19:06] <gregaf> don't think anybody's gotten that expensive yet
[19:07] * samppah (hemuli@namibia.aviation.fi) Quit (Ping timeout: 480 seconds)
[19:08] <jtang> hrmm
[19:09] <jtang> was talking to my manager earlier about getting a few fusion-io cards to play with
[19:10] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[19:12] * chutzpah (~chutz@ has joined #ceph
[19:12] <gregaf> I suspect you'll learn about the software IO limits if you go there, but it would be fun!
[19:13] <iggy> there was some talk on the kvm list about performance with them, might be worth a read
[19:14] <jtang> its really just for fun
[19:14] <jtang> plus it gives us a chance to annoy some of our sales reps to loan us toys to play with
[19:15] <jtang> we're thinking of taking 4 nodes with 40gb ib in them as well
[19:15] <iggy> heh, good ole sales guys
[19:15] <jtang> we'll be limited by ipoib or srp
[19:16] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:17] <iggy> done much testing with ipoib?
[19:18] * Kioob`Taff1 (~plug-oliv@local.plusdinfo.com) Quit (Read error: Operation timed out)
[19:20] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) Quit (Ping timeout: 480 seconds)
[19:20] <jtang> iggy: not much (nto recently anyway) and certainly not with ceph
[19:24] <joshd> timmclaughlin: jskinner: see http://ceph.com/docs/master/rados/operations/troubleshooting-osd/#stuck-placement-groups
[19:25] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) has joined #ceph
[19:30] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[19:31] * jskinner (~jskinner@ Quit (Remote host closed the connection)
[19:34] * sjustlaptop (~sam@68-119-138-53.dhcp.ahvl.nc.charter.com) has joined #ceph
[19:40] * fmarchand2 (~fmarchand@ has joined #ceph
[19:40] * jskinner (~jskinner@ has joined #ceph
[19:40] <fmarchand2> hi!
[19:40] <fmarchand2> hi slang !
[19:41] <slang> fmarchand2: hi
[19:41] <fmarchand2> how are you ?
[19:41] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[19:41] * tryggvil_ (~tryggvil@rtr1.tolvusky.sip.is) Quit ()
[19:42] <fmarchand2> slang : did you have time to check my log file ?
[19:43] <slang> fmarchand2: I'm good
[19:44] <slang> fmarchand2: I looked at it, but because it doesn't have the log from when the problem occurred, it turns out not to be that useful
[19:44] <slang> fmarchand2: you are still seeing that directory can't be removed, I assumed?
[19:44] <slang> assume*
[19:44] <fmarchand2> yup
[19:45] <fmarchand2> It's fine I guess it's not tht critical
[19:46] <fmarchand2> slang : now my main pb is a memory one ... I have a mds process taking more and more memory day after day ... and I don't know if it's gonna stop eating my ram ...
[19:48] <fmarchand2> slang : I have a process every 2 nights retrieving a lot of files from internet and storing it in folders and gzip files on the ceph cluster
[19:48] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Ping timeout: 480 seconds)
[19:49] <fmarchand2> slang : I would like to know if there is a limit of size in memory for the mds process
[19:50] <fmarchand2> I thought it was the cche size but apparently it's not because I reduced it and it didn't change anything
[19:51] * MarkS_ is now known as MarkS
[19:52] <slang> fmarchand2: you tried to reduce mds cache size in the config?
[19:52] * adjohn (~adjohn@ has joined #ceph
[19:57] * mdrnstm (~mdrnstm@206-169-78-213.static.twtelecom.net) has joined #ceph
[20:01] <fmarchand2> slang : yes I did .... should I restart or something after ?
[20:01] <slang> fmarchand2: yes you need to restart the mds
[20:02] <fmarchand2> slang : I think I did it but I'm gonna retry
[20:05] * rweeks (~rweeks@wlan-clients-7089.sc12.org) has joined #ceph
[20:06] * BManojlovic (~steki@ has joined #ceph
[20:08] * samppah (hemuli@namibia.aviation.fi) has joined #ceph
[20:09] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[20:22] * dmick (~dmick@2607:f298:a:607:5144:73e6:7ad0:5110) has joined #ceph
[20:23] * ChanServ sets mode +o dmick
[20:26] * fmarchand2 (~fmarchand@ Quit (Ping timeout: 480 seconds)
[20:28] * noob2 (a5a00214@ircip2.mibbit.com) has joined #ceph
[20:29] <noob2> i think snapshot causes my monitor to kernel panic
[20:29] <noob2> it says fixing recursive fault but reboot is needed and hung
[20:30] <Psi-jack> Hmmm.. So yeah. 1 test vm running with rbd to ceph, and I see the kernel yelling about Disk I/O wait, just doing an apt-get update operation. :/
[20:31] <Psi-jack> Backed by 2 OSD's, and 3 Mon's over a 1Gbit ethernet network to the storage servers, dedicated to just storage traffic.
[20:32] <joshd> noob2: that's strange, snapshots aren't doing much interesting on the monitors
[20:32] <noob2> yeah i know
[20:32] <noob2> i suspected as much also
[20:32] <joshd> Psi-jack: do you have rbd caching enabled?
[20:32] <Psi-jack> joshd: Not sure?
[20:33] <joshd> it's disabled by default: http://ceph.com/docs/master/rbd/rbd-config-ref/
[20:33] <Psi-jack> I do see on the ceph osd's that shows it's slow at the OSD level: log [WRN] : slow request 32.268674 seconds old, received at 2012-11-15 13:36:19.931773: osd_op(client.12039.0:126168 rb.0.52bb.17cd8cb1.00000000025c [write 1576960~4096] 2.c271fe77) v4 currently waiting for sub ops
[20:33] <Psi-jack> 32 seconds is... Pretty big. ;)
[20:34] <Psi-jack> joshd: Do you happen to know how Proxmox utilizes rbd?
[20:34] <joshd> through qemu
[20:34] <noob2> joshd: i think the snap was created. when i tried to snap again it says it was there. i guess something else caused the monitor to crash
[20:34] <Psi-jack> joshd: So would this still help in that situation?
[20:35] <noob2> is there a list snaps command?
[20:36] <joshd> Psi-jack: yes, although the osd slow request warning should be figured out. it may be the osd bug that isn't removing an op from the tracking when it's completed, a few other folks have seen that happen
[20:37] <Psi-jack> joshd: Looking into it, I do have cache=writeback.
[20:37] <joshd> Psi-jack: with qemu >= 1.2?
[20:37] <Psi-jack> joshd: Yeaaaah.. That's more what I'm thinking. And it's 1.2.0
[20:37] <joshd> noob2: for rbd, it's 'rbd snap ls <imagename>'
[20:38] <noob2> yeah i snapped a pool instead
[20:39] <joshd> noob2: not sure about that, pool snapshots are much less used
[20:39] <noob2> ok
[20:40] <noob2> as far as object storage goes let me see if i have this right. i create a pool named whatever, add a user to that pool. do i then add a rbd to that pool and set permissions to allow the user access to it?
[20:40] <noob2> they get their access ticket through the gateway and then are granted access to the rbd container
[20:41] <Psi-jack> joshd: My OSD storage itself is actually housed on 2 Dell PowerEdge 830's with SAS RAID-10 drives, 4-drives each server. The filesystem itself on those is ZFS, but I made a subvolume block that I put ext4 onto and use that for the OSD storage.
[20:41] <joshd> Psi-jack: try doing 'ceph -w' in one terminal and 'ceph osd tell \* bench' in another. results will go to the ceph -w output. that'll tell you if one of your osds is slow
[20:41] <noob2> raid-10 is generally pretty quick. what kind of caching does that controller have?
[20:42] <joshd> noob2: it sounds like you're mixing radosgw and rbd terms together there. they're entirely separate access methods
[20:42] <noob2> ok
[20:42] <noob2> yeah i'm a little confused how the gatewy works
[20:42] <Psi-jack> It's a battery-backed PERC. One sec. ;)
[20:43] <noob2> if i have a dept that i want to give object storage to, how would i go about that?
[20:43] <noob2> at a high level i guess
[20:43] <joshd> the gateway basically translates s3 or swift HTTP requests to librados
[20:43] <noob2> ok
[20:43] <Psi-jack> 128 MB cache
[20:44] <noob2> Psi-jack are your writes to the cluster smaller (4K?)
[20:44] <joshd> the gateway has its own layer of users and access keys needed by the s3 and swift protocols
[20:44] <noob2> i noticed pretty bad performance when i had writes that small on my sas drives
[20:44] <noob2> joshd: ok so that gateway is its own thing
[20:44] <Psi-jack> noob2: Not entirely sure. This is 1 singular qemu-rbd disk storage for an Ubuntu 12.04 guest OS running on it.
[20:45] <noob2> i thought the gateway was just a pass through kind of deal to rados objects
[20:46] <noob2> joshd: thanks for the clarification :)
[20:46] <Psi-jack> joshd: I'm not seeing anything in the ceph -w that would describe slowness?
[20:46] <joshd> Psi-jack: the bench results should appear there
[20:46] <noob2> Psi-jack: ceph -w just follows the logging output of ceph
[20:46] <noob2> and that ^ ^
[20:46] <Psi-jack> Ahhh
[20:46] <noob2> you give it a minute and it'll output a bench result
[20:46] <Psi-jack> I was seeing pgmap stuff, now I just saw bench. ;)
[20:47] <noob2> great
[20:47] <noob2> what are the results?
[20:47] <Psi-jack> Unfortunately, I hit bench a few times while waiting for something to show up. Thinking the pgmap lines were part of the "response"
[20:47] <noob2> ah
[20:47] <noob2> ok
[20:47] <Psi-jack> 2012-11-15 14:47:30.123255 osd.1 [INF] bench: wrote 1024 MB in blocks of 4096 KB in 77.013002 sec at 13615 KB/sec
[20:48] <noob2> yeah that's a bit slow
[20:48] <noob2> i was seeing around 30MB/s
[20:48] <noob2> on my sas drives
[20:48] <noob2> are your journals housed on the same disk?
[20:49] <Psi-jack> It's a /bit/ slow/ yes. These servers can push about 60 MB/s or so, but they are being used by other servers as well. The NFSv4->qcow2 servers.
[20:49] <Psi-jack> Yes
[20:49] <noob2> ok
[20:50] <Psi-jack> Heh, bench is getting slower.. And slower...
[20:50] <Psi-jack> Here was the first one to come back: 2012-11-15 14:46:12.801200 osd.1 [INF] bench: wrote 1024 MB in blocks of 4096 KB in 54.926261 sec at 19090 KB/sec
[20:50] <Psi-jack> 20 MB/s, is reasonable.
[20:51] * houkouonchi-work (~linux@ has joined #ceph
[20:51] <Psi-jack> Almost finally to the last bench. 6 times. ;)
[20:51] <noob2> when i first started on the ceph adventure i was expecting 100MB/s lol
[20:51] <Psi-jack> And it's getting back up to speed.
[20:51] <noob2> i think the speed comes from scaling wide
[20:51] <Psi-jack> Usually. ;)
[20:51] <noob2> and each server gives what it can
[20:52] <Psi-jack> And scaling over infiniband. ;)
[20:52] <noob2> yeah that wouldn't hurt either :D
[20:53] <Psi-jack> OKay. All bench's finished. Doing one more to get the 1 result. ;)
[20:53] <noob2> nice
[20:53] * nhorman (~nhorman@nat-pool-rdu.redhat.com) Quit (Quit: Leaving)
[20:54] <noob2> joshd: i see on the wiki page now. All the gateway stuff is done via radosgw-admin
[20:54] <noob2> pool creation, users, etc
[20:54] <joshd> yeah, don't trust the wiki though, I'm pretty sure the gateway stuff there is old
[20:54] <Psi-jack> 2012-11-15 14:53:50.820516 osd.1 [INF] bench: wrote 1024 MB in blocks of 4096 KB in 51.204446 sec at 20478 KB/sec
[20:54] <Psi-jack> So, roughly, consistent 20 MB/s capability.
[20:55] <noob2> so scale out to 10 servers and you have a pretty fast cluster haha
[20:55] <noob2> i'm hoping to start with 6 servers and see where it goes
[20:55] <joshd> putting your journals on separate disks helps a lot, since all your writes are going to the journals too
[20:56] <noob2> i haven't decided on 4x1Gb/s connections or 1 10Gb/s connection. i'm waiting on a networking change to change the xmit-hash-policy on the switches to allow all the connections
[20:56] <Psi-jack> But, still. The main thing is, I'm getting /serious/ wait times from my 1 singular qemu-rbd client facilitating this.
[20:56] <Psi-jack> noob2: 1 10Gb/sec
[20:56] <noob2> joshd: i agree. i saw the same performance jump. it was almost double
[20:56] <noob2> so you have 1 10Gb/s connection?
[20:56] <Psi-jack> No. I have just 1x1Gb. :)
[20:57] <noob2> oh ok
[20:57] <Psi-jack> BUt, that's in my home. ;)
[20:57] <noob2> lol
[20:57] <noob2> i'm thinking 4x1Gb will be plenty after seeing the rados benchmarks
[20:57] <joshd> Psi-jack: I think apt-get may be doing syncs, which will make the caching not help
[20:58] <noob2> so on your vm you're seeing io waits from iostat?
[20:58] <Psi-jack> Yes
[20:58] <noob2> are your servers colocated with others on the switch?
[20:58] <noob2> maybe your problem is upstream?
[20:58] <Psi-jack> I have two distinct networks, actually.
[20:59] <noob2> try installing dstat on your osd's. that usually gives me a lot of info quickly
[20:59] <Psi-jack> 1x1Gb to the storage network, 1x1Gb to the main network.
[20:59] <Psi-jack> So storage has it's own dedicated network.
[20:59] <noob2> dstat -N bond0 -ndl --disk-util
[20:59] <noob2> ok i see
[20:59] <noob2> that's good
[20:59] <joshd> Psi-jack: I wonder if you have enough pgs to distribute to all osds?
[21:00] <noob2> yeah i was going there next
[21:00] <Psi-jack> joshd: I don't know what that means. ;)
[21:00] <noob2> i upped mine from the default to 1200
[21:00] <noob2> pg's are placement groups
[21:00] <noob2> shards of storage i guess you could say
[21:00] <Psi-jack> Okay. And where would I change this? ;)
[21:00] <noob2> with enough pg's it'll balance the load over all your osd's evenly
[21:00] <noob2> when you create the pool
[21:00] <noob2> i don't know if you can change it after it is made
[21:01] <Psi-jack> Ahhhh..
[21:01] <noob2> joshd surely knows :)
[21:01] <joshd> not yet, but soon :)
[21:01] * Psi-jack chuckles.
[21:01] <noob2> haha
[21:02] <Psi-jack> So I take it it goes into the [osd] section of ceph.conf before mkcephfs?
[21:02] <noob2> i don't think so? i was able to do it with a pool creation command
[21:02] <noob2> ceph osd pool create {pool-name} {pg-num} [{pgp-num}]
[21:02] <Psi-jack> Hmmm..
[21:02] <Psi-jack> I never once used that.
[21:03] <noob2> i think i did ceph osd pool create test 1200
[21:03] <noob2> i don't know how pg-num differs from pgp-num
[21:03] <joshd> mkcephfs creates the default pools (data metadata and rbd) for you
[21:03] <Psi-jack> Ahhh. ;)
[21:03] <noob2> ceph osd pool get {pool-name} {key}
[21:03] <noob2> if you issue that command it'll tel you how many pg groups you have
[21:04] <noob2> so ceph osd pool get test-pool pg-num
[21:04] <noob2> the wiki says about 100 pg's per OSD is recommended
[21:04] <Psi-jack> ceph osd pool get rbd pg-num: don't know how to get pool field pg-num
[21:05] <noob2> :(
[21:05] <noob2> got it
[21:05] <noob2> pg_num
[21:05] <noob2> not pg-num
[21:06] <Psi-jack> 128.
[21:06] <noob2> ok
[21:06] <noob2> i think that is the default
[21:06] <noob2> how many osd's did you have again?
[21:06] <noob2> 2?
[21:06] <Psi-jack> Yes
[21:06] <noob2> try making another one with 200 or 400 pg's
[21:07] <Psi-jack> I'm seeing how to change the pg_num value now..
[21:07] <Psi-jack> Same command, but instead of get, it's set. :)
[21:07] <Psi-jack> Ahhh
[21:07] <Psi-jack> But broken implementation. ;)
[21:07] <noob2> correct
[21:07] <noob2> it iddn't work
[21:07] <Psi-jack> heh
[21:08] <noob2> here's what happened when i put the journal on ssd's: 2012-11-15 15:07:21.568277 osd.1 [INF] bench: wrote 1024 MB in blocks of 4096 KB in 16.395422 sec at 63955 KB/sec
[21:08] <Psi-jack> Well, blasted... Well, I'll try re-creating it. After I figure out how to do it, when I get home tonight, though I'll be blowing away this ceph all over again and the 1 VM I installed. (thankfully, just 1 VM)
[21:09] <noob2> yeah
[21:09] <noob2> i think you can get the same benefit from moving the journal to a spinning disk
[21:09] <noob2> since my pci ssd's are insanely overkill
[21:09] <noob2> they read/write at like 2GB/s
[21:09] <Psi-jack> Heh yeah.
[21:10] <Psi-jack> Eventually I should get SSD's for these servers.
[21:10] <Psi-jack> If nothing else than for the journals.
[21:10] <noob2> yeah even a small ssd with partitions would help
[21:11] <noob2> according to dstat i'm maxing out the disk
[21:12] <noob2> it writes the bench at 300MB/s until it fills the journal and then it drops to 50MB/s
[21:12] * Leseb (~Leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) has joined #ceph
[21:12] <noob2> weird thing is i have a 10GB journal
[21:12] <Psi-jack> Hmmm, so can't change pg_num with mkcephfs approach, I see?
[21:12] <Psi-jack> As in, can't set the pg_num value for what it's doing?
[21:13] <noob2> i don't believe so
[21:13] <joshd> I think you can in ceph.conf actually
[21:13] <noob2> ok cool
[21:13] <joshd> it's only used for mkcephfs though
[21:14] <noob2> joshd: can you set the internal for how often the journal is flushed to disk?
[21:16] <noob2> i found it
[21:16] <noob2> yes you can set the journal sync interval
[21:16] * fmarchand2 (~fmarchand@85-168-75-42.rev.numericable.fr) has joined #ceph
[21:16] <joshd> I don't see that option...
[21:17] <noob2> http://ceph.com/docs/master/rados/configuration/journal-ref/
[21:17] * buck (~buck@bender.soe.ucsc.edu) Quit (Quit: Leaving.)
[21:17] <noob2> default seems to be 5 seconds
[21:17] <noob2> that jives well with what i'm seeing over dstat
[21:18] <noob2> it'll zoom along for 5 seconds and then die when it starts flushing to disk haha
[21:18] <noob2> cause my ssd's are way faster than the sas drives
[21:18] <joshd> yeah, that makes sense
[21:18] <noob2> that's cool
[21:18] * aliguori (~anthony@ Quit (Ping timeout: 480 seconds)
[21:18] <noob2> based on network bandwidth i can calculate what size journal and also how long of a commit i can stand before flushing
[21:18] <noob2> juice the numbers :D
[21:23] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[21:23] * pedahzur (~jkugler@216-67-98-32.static.acsalaska.net) has joined #ceph
[21:24] <pedahzur> There is still a link in Google to ceph.com/wiki/Monitoring. This page no longer exists. Is there a way to tell where the content was moved? Or dig up the deleted page?
[21:24] * fmarchand2 (~fmarchand@85-168-75-42.rev.numericable.fr) Quit (Ping timeout: 480 seconds)
[21:25] <pedahzur> Hmm...lots of broken links in google.
[21:25] <pedahzur> Is ceph.com not reporting 404's correctly? Maybe doing 302's instead?
[21:26] <noob2> i think they reorganized the wiki
[21:27] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) Quit (Quit: Leaving.)
[21:28] * buck (~buck@bender.soe.ucsc.edu) has joined #ceph
[21:28] <rweeks> the wiki has been deprecated
[21:29] <rweeks> if you are missing info point it out to rturk or scuttlemonkey
[21:31] <scuttlemonkey> yeah, our doc has had quite a bit of love lately...and there was just too much old/outdated stuff on the wiki
[21:32] <scuttlemonkey> all /wiki/ links should be mod_rewriting to /docs/master/ now
[21:32] <pedahzur> scuttlemonkey: OK, gotcha...but IIRC, a rewrite (30x) doesn't tell google to remove it from its search cache. There are links from 2010 in their result set that no longer exist.
[21:33] <scuttlemonkey> well I'm doing the rewrite via 301
[21:33] <scuttlemonkey> giving the google index time to xfer some of the previous juice
[21:33] <scuttlemonkey> then we'll put the kaibosh on it
[21:33] <pedahzur> This one generates a 404: http://ceph.com/docs/master/cluster-ops/monitoring/
[21:33] <pedahzur> Not sure if it's supposed to be there or not.
[21:34] <scuttlemonkey> although, admittededly my google-fu may be outdated
[21:34] <scuttlemonkey> hmm, dunno
[21:34] <scuttlemonkey> I didn't touch any link w/in /docs/
[21:35] <scuttlemonkey> where did you grab that link?
[21:36] * sagewk (~sage@2607:f298:a:607:91b3:f084:6ac7:d6a4) has joined #ceph
[21:37] <Psi-jack> Still trying to find that pgs ceph.conf setting. ;)
[21:37] <pedahzur> scuttlemonkey: That was a google result for ceph monitoring
[21:38] <Psi-jack> I do see osd pg bits, though?
[21:38] <scuttlemonkey> pedahzur: ahh, John must have moved things around in his doc work
[21:39] <scuttlemonkey> I'll poke him about adding some rewrites for his changes
[21:39] <scuttlemonkey> cuz I don't see the cluster ops stuff anywhere
[21:39] <Psi-jack> But these don't seem to match up to what I got, 128...
[21:41] <Psi-jack> joshd: Any idea how to set the pg_num mkcephfs will use?
[21:41] <Psi-jack> In the mkcephfs script I don't even see anything about pg_num or pg
[21:42] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[21:42] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit ()
[21:45] * sagewk (~sage@2607:f298:a:607:91b3:f084:6ac7:d6a4) Quit (Remote host closed the connection)
[21:48] * fmarchand2 (~fmarchand@ has joined #ceph
[21:49] <fmarchand2> slang : thank you I changed the cache size and it worked ... I used the wrong setting for the mds. So I freed 3gb of RAM !
[21:53] * danieagle (~Daniel@ has joined #ceph
[21:54] <slang> fmarchand2: cool
[21:55] * eternaleye (~eternaley@tchaikovsky.exherbo.org) Quit (Remote host closed the connection)
[21:56] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[21:56] * loicd (~loic@magenta.dachary.org) has joined #ceph
[21:57] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) has joined #ceph
[21:59] * noob2 (a5a00214@ircip2.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[22:01] <dmick> Psi-jack: pgnum is set at pool creation time, but the default pools don't have any control over that IIRC
[22:01] <dmick> you can delete and recreate the pools if they're empty
[22:01] <dmick> ceph osd pool create <pool> <pg_num> [<pgp_num>]
[22:04] * eternaleye (~eternaley@tchaikovsky.exherbo.org) has joined #ceph
[22:04] <scuttlemonkey> pedahzur: fwiw you should be able to get the old cluster-op stuff from here: http://ceph.com/docs/master/rados/operations/
[22:04] <scuttlemonkey> w/ a couple exceptions
[22:05] <scuttlemonkey> I'll help John with some rewrite magic tomorrow
[22:05] * jjgalvez (~jjgalvez@cpe-76-175-17-226.socal.res.rr.com) has joined #ceph
[22:08] <dmick> Psi-jack: it's actually osdmaptool that does it
[22:08] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[22:08] <dmick> and that tool can take --pg-bits arguments, but there's no interface for passing it in from mkcephfs
[22:08] * aliguori_ (~anthony@ has joined #ceph
[22:08] * tryggvil_ (~tryggvil@15-80-126-149.ftth.simafelagid.is) has joined #ceph
[22:08] <joshd> dmick: I thought you could set pg_bits in ceph.conf and it would use that
[22:09] <dmick> uhhh...maybe\
[22:10] <dmick> yeah, those locals in osdmaptool are initialized from the conf file values, true
[22:13] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[22:14] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) Quit ()
[22:14] * fmarchand2 (~fmarchand@ Quit (Ping timeout: 480 seconds)
[22:14] <jefferai> are the rbd/libceph kernel modules pretty stable?
[22:14] <jefferai> not for cephfs, but ganeti wants to access RBD via that instead of via rados
[22:14] <jefferai> like, instead of just using the pool directly
[22:15] * tryggvil (~tryggvil@16-80-126-149.ftth.simafelagid.is) Quit (Ping timeout: 480 seconds)
[22:15] * tryggvil_ is now known as tryggvil
[22:16] * houkouonchi-work (~linux@ Quit (Read error: Connection reset by peer)
[22:17] <rweeks> yes, jefferai
[22:18] <jefferai> okay
[22:18] <jefferai> I remember hearing that the kernel client for cephfs wasn't too stable so wasn't sure about rbd
[22:18] <jefferai> it's too bad ganeti does it this way, though, as it'd be much nicer for it to use librados rather than me having to allocate images of particular sizes
[22:19] <jefferai> like what opentack does
[22:19] <rweeks> right
[22:19] <rweeks> a lot of work went into rbd for argonaut
[22:20] <rweeks> and now a lot of work is going into cephfs for bobtail
[22:20] <jefferai> ah, cool
[22:20] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Ping timeout: 480 seconds)
[22:21] <pedahzur> scuttlemonkey: Thanks!
[22:21] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[22:22] * denken (~denken@dione.pixelchaos.net) Quit (Quit: migration)
[22:24] <joshd> jefferai: just make you have a new enough kernel - 3.6 is recommended
[22:25] * houkouonchi-work (~linux@ has joined #ceph
[22:27] <gregaf> rweeks: you're running a bit early — Bobtail is frozen and has some CephFS work (but much more RBD work), and hopefully Cuttlefish will have a noticeably better CephFS
[22:27] <rweeks> fair enough
[22:27] <rweeks> but good to know.
[22:28] * tryggvil_ (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[22:28] <jefferai> joshd: woah
[22:28] <jefferai> okay
[22:28] <jefferai> um, was using the recommended Ubuntu Precise
[22:29] <joshd> jefferai: check out http://ceph.com/docs/master/install/os-recommendations/
[22:29] <jefferai> ah, the "Ceph Kernel Client"
[22:29] <jefferai> I thought that was just for cephfs
[22:30] <joshd> there were a bunch of fixes to libceph (used by both kernel rbd and cephfs) in 3.6
[22:30] <jefferai> I see
[22:30] <jefferai> so what do you guys use, then?
[22:30] <jefferai> Precise with updated kernels?
[22:31] * thingee (~thingee@ps91741.dreamhost.com) has joined #ceph
[22:31] <joshd> yeah, when we run the kernel client
[22:31] <jefferai> and if I did that, then it would be safe(?) to use btrfs for the OSDs instead of XFS
[22:32] * tryggvil_ (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Read error: Connection reset by peer)
[22:32] <joshd> much safer than 3.2
[22:32] <jefferai> do you get your kernels from a specific place? Or find some daily kernel from the kernel-ppa that works?
[22:32] * tryggvil_ (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[22:32] <joshd> we build packages of each branch in our kernel tree
[22:33] * thingee (~thingee@ps91741.dreamhost.com) Quit ()
[22:33] <joshd> http://gitbuilder.sepia.ceph.com/gitbuilder-precise-kernel-amd64/
[22:34] <joshd> output e.g. here: http://gitbuilder.ceph.com/kernel-deb-precise-x86_64-basic/ref/linux-3.6.3/
[22:34] * tryggvil (~tryggvil@15-80-126-149.ftth.simafelagid.is) Quit (Ping timeout: 480 seconds)
[22:34] * tryggvil_ is now known as tryggvil
[22:35] <jefferai> joshd: awesome
[22:35] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[22:35] <jefferai> thanks
[22:36] <jefferai> so just to make sure I understand things: on the storage boxes you generally recommend sticking with your packages on a default precise base; you would only recommend the newer kernel if using the kernel client?
[22:36] <jefferai> or do you recommend the newer kernel on the storage boxes as well?
[22:37] <joshd> newer kernel is only needed for osds if you're using btrfs, the default one is fine otherwise
[22:39] <jefferai> do you recommend the newer kernel + btrfs, or older kernel + xfs
[22:39] <joshd> xfs
[22:39] <jefferai> okay
[22:39] <jefferai> sounds good :-)
[22:40] <Psi-jack> dmick: Hmmm.
[22:41] <jefferai> joshd: thanks
[22:41] <joshd> np
[22:41] <Psi-jack> dmick: Well, mine's not empty, but nothing really important /in/ them. i just did the basic mkcephfs approach.
[22:43] <dmick> syncfs: is there a way we can use syscall() to get around these silly libc limitations?
[22:43] <Psi-jack> This "ceph" tool seems to be VERY negligent in any form of docs.. I can't even find anything of what I can do with it. ;)
[22:43] <dmick> ceph.com/docs is the best answer I know of right now
[22:45] * flesh (547908cc@ircip1.mibbit.com) has joined #ceph
[22:45] <flesh> hello
[22:46] <flesh> I am trying Ceph, with just a single node configuration, and I am having some trouble
[22:47] <flesh> I used the 5 minute start
[22:48] <flesh> and when I do the mkcephfs it says
[22:48] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:48] * Leseb (~Leseb@5ED17881.cm-7-2b.dynamic.ziggo.nl) Quit (Quit: Leseb)
[22:49] <flesh> mkcephfs: line 119: .0 + 1: sintax error: operand expected (error token is ".0 + 1")
[22:49] <flesh> any idea?
[22:50] <flesh> I realize the ".0" complain is about the "[osd.0]"
[22:50] <flesh> but I wrote it just like it is written on the config example
[22:51] <Psi-jack> dmick: Yeah.. But, how do I show what "pools" I have?
[22:51] <sjustlaptop> flesh: can you put your ceph.conf on pastebin?
[22:51] <sjustlaptop> or something
[22:51] <flesh> sure
[22:52] <dmick> Psi-jack: rados lspools
[22:52] <jstrunk_> How should I tune ceph to properly distribute data to different sized OSDs?
[22:52] <Psi-jack> Hmmm, cool!
[22:53] <flesh> http://pastebin.com/2A33DFQV
[22:53] <Psi-jack> rados actually looks better documented.. For what it does. ;)
[22:53] <jstrunk_> I have 3 servers. Two have 2 1.8T disks as OSDs, and the third has 3 275GB OSD disks.
[22:53] * MikeMcClurg (~mike@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[22:53] * aliguori_ (~anthony@ Quit (Remote host closed the connection)
[22:53] <Psi-jack> dmick: So I just need to delete my rbd pool (i presume?), and re-create it with a different pgs?
[22:53] <dmick> Psi-jack: that should work
[22:54] <jstrunk_> The server with 3 OSDs has only been using two of them.
[22:55] * rweeks (~rweeks@wlan-clients-7089.sc12.org) Quit (Quit: ["Textual IRC Client: www.textualapp.com"])
[22:55] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit (Quit: Leaving.)
[22:55] <jstrunk_> After testing this for a few days, 2 of the OSDs on the server with 3 OSDs are full, but the 3rd OSD is empty.
[22:55] <jefferai> jstrunk_: custom crushmap?
[22:56] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[22:56] * miroslav (~miroslav@wlan-clients-8094.sc12.org) Quit ()
[22:56] <dmick> flesh: my mkcephfs and ceph_common.sh don't contain a +, and it probably didn't say "sintax"; can you show the exact command you gave and its exact output?
[22:56] <Psi-jack> dmick: I see, cool, yeah, I setup my proxmox ve to use pool named rbd, which is what it did on ceph, created the pool when I first used it, I suppose.
[22:57] <dmick> Psi-jack: you can certainly create other pools too
[22:57] <dmick> nothing magic about rbd other than 'the default for the cli tools'
[22:57] <flesh> dmick there you are http://pastebin.com/16GsFDaY
[22:58] <joshd> jstrunk_: sound like you might not have enough pgs: http://ceph.com/docs/master/rados/operations/placement-groups/
[22:58] <Psi-jack> dmick: That's interesting. :)
[22:59] <jstrunk_> joshd: ceph status says I have 1536 pgs
[22:59] <dmick> flesh: which ceph version is this?
[22:59] <wer> "The Ceph community provides a slightly optimized version of the apache2 and fastcgi packages." Where can I actually find these?
[22:59] <jstrunk_> joshd: sorry, there are 512 pgs for the pool I am using
[22:59] <flesh> dmick: ceph-0.48.2argonaut
[23:00] <joshd> jstrunk_: could you pastebin the output of 'ceph osd tree'?
[23:01] <jstrunk_> joshd: http://pastebin.com/XHGRecvT
[23:01] <jstrunk_> osd.0 is on dell10
[23:04] <joshd> jstrunk_: do 'ceph osd crush set osd.0 1 pool=default rack=unknownrack host=dell10'
[23:04] <joshd> jstrunk_: the problem is that it's not in the crush hierarchy
[23:04] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) Quit (Remote host closed the connection)
[23:05] <joshd> jstrunk_: you'll also want to adjust the weights of your osds to be proportional to their capacity, since you have different size disks
[23:05] * Cube (~Cube@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[23:05] <dmick> flesh: I am mystified. I don't know what line 119 is referring to. Maybe try bash -x mkcephfs to isolate the problem?
[23:05] * PerlStalker (~PerlStalk@perlstalker-1-pt.tunnel.tserv8.dal1.ipv6.he.net) has joined #ceph
[23:05] <jstrunk_> joshd: thank you
[23:07] <joshd> wer: http://gitbuilder.ceph.com/libapache-mod-fastcgi-deb-precise-x86_64-basic/ref/master/
[23:08] <wer> thank you joshd. I think I am running squeeze.
[23:09] <flesh> dmick: oh yes... now I see. It cannot find some binaries
[23:10] <flesh> I must change PATH and LD_LIBRARY_PATH I guess
[23:10] <flesh> dmick: thanks :) I see if I can fix it
[23:10] <wer> joshd: I am trying you account for a 500 error I get from fastcgi... puts are successful... but clients see the error through the radosgw. Any of that sound familiar to you?
[23:11] <joshd> wer: not to me, but maybe to yehudasa
[23:13] <wer> It sucks because everything appears to be working fine... I am gettign a 500 error. Duplicate status is being returned to fastcgi. I think that is what is really happening...
[23:16] <Psi-jack> Hmmm, still seems to be going rather slow, even with 400 pgs. So far, anyway.
[23:16] <joshd> wer: I'd check radosgw and apache's logs, maybe turn up radosgw debugging if there's no clue there
[23:16] <dmick> flesh: yw. What couldn't it find?
[23:18] <wer> /var/www/radosgw.fcgi" aborted: error parsing headers: duplicate header 'Status'
[23:18] <wer> That is from apache error.log
[23:20] <yehudasa> wer: which fastcgi module are you running?
[23:20] <wer> libapache2-mod-fastcgi ?
[23:20] <yehudasa> which version?
[23:20] <wer> 2.4.6-1
[23:21] <yehudasa> first, you can try setting 'rgw print continue = false', see if it fixes the issue
[23:21] <wer> nice! will try.
[23:21] <yehudasa> then, you can install the modified fastcgi module that supports 100-continue
[23:22] <wer> yehudasa: yeah... that is what I was suspecting I needed... but I will try the first try. Does that go in my ceph.conf btw?
[23:23] <yehudasa> wer: yeah
[23:24] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) Quit (Quit: Leaving.)
[23:26] * Cube (~Cube@ has joined #ceph
[23:27] <wer> yehudasa: I think that actually got rid of it :) ty. Now my client fails on the 200 :) any way to force a 201 instead?
[23:28] <wer> your fix helped s3libs with an xml parsing error too :)
[23:29] <yehudasa> wer: yeah, that fixes it but you have 100-continue disabled
[23:30] <yehudasa> (apache will send 100 response automatically, disregarding if the PUT request is going to succeed or not)
[23:30] * sagewk (~sage@2607:f298:a:607:91b3:f084:6ac7:d6a4) has joined #ceph
[23:32] <wer> hmm. ok. ahh I gotcha so now I NEED fastcgi that supports 100's. for squeeze
[23:34] <yehudasa> yeah, just note that it's not going to help with 200 vs 201
[23:35] <wer> yehudasa: Is there any hope on that? That is going to be my likely next hurdle I think.
[23:38] <yehudasa> wer: not really, unless you want to compile rgw from source and modify that
[23:38] * aliguori (~anthony@cpe-70-123-145-75.austin.res.rr.com) has joined #ceph
[23:38] <yehudasa> wer: looking at the s3 api, PUT of object returns 200
[23:39] <wer> :) ok. I'll have to dig a little further. My current object store seems to return a 201. And the puts interpret a 200 as a fail. perhaps I can fix that on the client side.
[23:40] <yehudasa> well, 200 is a success response, so yeah
[23:40] * jlogan (~Thunderbi@2600:c00:3010:1:1ccf:467e:284:aea8) Quit (Ping timeout: 480 seconds)
[23:41] <slang> sagewk: just submitted #3498
[23:43] * miroslav (~miroslav@wlan-clients-8094.sc12.org) has joined #ceph
[23:44] * Kioob (~kioob@luuna.daevel.fr) has joined #ceph
[23:45] <wer> thanks yehudasa. Just to clarify for myself, I need to find a fastcgi that handles 100's to be safe at this point right?
[23:45] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[23:45] * loicd (~loic@magenta.dachary.org) has joined #ceph
[23:47] <yehudasa> wer: you need a modified fastcgi module, yes. This one: https://github.com/ceph/mod_fastcgi
[23:48] <wer> perfect. Thanks again for the help!
[23:49] <flesh> dmick: I had an old version of mkceph installed, and the PATH was not updated.. so there was come versions confusion. But after that it wasn't still working, and I could see that I needed to call mkcephfs from /etc/init.d/ ...
[23:49] <flesh> *some versions confusion
[23:49] <dmick> ok. glad it's working
[23:53] <wer> oh hey yehudasa. I am not sure, but for http 1.0 the correct response may be 201. I need to check on that for sure. My client is speaking http 1.0.
[23:55] * jskinner (~jskinner@ Quit (Remote host closed the connection)
[23:57] <Psi-jack> Interesting.
[23:58] <Psi-jack> Well, with pg_num of 400, I am getting less issues, and it is actually a bit faster.
[23:58] <Psi-jack> Instead of taking hours to install Ubuntu 12.04 server into a VM on rdb-ceph, it's taking about an hour. :)
[23:59] * timmclaughlin (~timmclaug@ Quit (Read error: Operation timed out)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.