! !
White!Paper!
!
NexentaStor!Solutions!Guide!for!!
Mirantis!OpenStack!
!
Version!1.0!!
October!2014!
!
!
Table!of!Contents!
Preface'...'3! About'this'document'...'3! Audience'...'3! External'References'...'4! Document'History'...'4! Overview'...'5! Reference'Architecture'...'6! Node'configuration'...'7! High'Availability'configurations'...'7! Network'topologies'...'7! Installation'and'Configuration'...'8! Software'requirements'...'8! Prerequisites'...'8! Configure'OpenStack'Glance'to'store'Images'on'NexentaStor'...'9! Export'the'newly'created'folder'through'NFS'...'10! Configure'OS'to'mount'NexentaStor'file'system'...'11! Updating'the'Glance'configuration'...'11! Configure'OpenStack'Cinder'to'use'ISCSI'from'NexentaStor'for'block'device'storage'...'12! Configure'Cinder'to'use'the'NexentaStor'ISCSI'backend'driver'...'12! Configure'OpenStack'Cinder'to'use'NFS'from'NexentaStor'for'block'device'storage'...'13! Export'the'newly'created'folder'through'NFS'...'14! Update'the'Cinder'driver'for'NexentaStor'4.0.x'...'15! Configure'Cinder'to'use'NexentaStor'NFS'Driver'...'15! Validating'the'installation'...'17! ! ! ! ! !!
Preface!
! About!this!document! This!reference!architecture!describes!how!to!integrate!Mirantis!OpenStack!5.1!(using!OpenStack!Icehouse)! with!NexentaStor!4.0.x,!utilizing!NexentaStor!as!a!backend!storage!for!both!golden!images!as!well!as!virtual! machine!storage.!! ! Audience! This!document!is!intended!for!system!administrators!that!want!to!integrate!NexentaStor!with!an!existing! OpenStack!installation.! It!is!expected!that!the!reader!is!experienced!with!OpenStack!and!has!basic!knowledge!of!NexentaStor! ! ! !! External!References!! ! For!additional!information,!please!see!the!following!documents:! Reference! Location! Nexenta!website! www.nexenta.com! Mirantis!website! www.mirantis.com! OpenStack!website! www.openstack.org! NexentaStor!Download!and!Documentation! http://nexenta.com/products/downloads/nexentastor!! Mirantis!OpenStack!5.1!download! http://software.mirantis.com! OpenStack!Cinder!Driver!Reference!
documentation!for!Nexenta!Driver! http://docs.openstack.org/icehouse/configVreference/content/nexentaVdriver.html!
Nexenta!ISCSI!driver!configuration!options! http://docs.openstack.org/icehouse/configVreference/content/nexentaViscsiVdriverVoptions.html!! Nexenta!NFS!driver!configuration!options! http://docs.openstack.org/icehouse/configVreference/content/nexentaVnfsVdriverVoptions.html!! GitHub!repository!for!Nexenta!Cinder!
Drivers! https://github.com/openstack/cinder/tree/master/cinder/volume/drivers/nexenta!!
!
!
Document!History!
!
Version! Date! Changes!
1.0! October,!2014! Initial!version!
!
!
Overview!!
! NexentaStor!is!a!SoftwareVDefinedVStorage!platform!that!delivers!unified!file!and!block!storage!services! scalable!into!the!petabyte!range!using!industry!standard!hardware,!and!includes!advanced!data!management! capabilities.! When!deployed!in!OpenStack!environments!NexentaStor!provide!a!cost!effective!and!scalable!backend!for! storing!both!virtual!machine!images!and!application!data.! The!integration!drivers!between!NexentaStor!and!OpenStack!have!been!a!part!of!the!upstream!releases!of! OpenStack!since!the!Essex!release!in!2012.!With!the!integration!in!the!upstream!Cinder!drivers!NexentaStor! works!‘out!of!the!box’!with!all!major!distributions!of!OpenStack,!including!Mirantis!OpenStack!5.1! ! NexentaStor!can!provide!storage!for!OpenStack!in!a!number!of!ways:! • Storage!for!the!Image!Repository!service!(Glance)!through!NFS! • Block!Storage!for!the!Cinder!Volume!service!through!both!ISCSI!and/or!NFS! • Application!data!storage!through!both!ISCSI!and!NFS! ! Below!is!a!conceptual!architecture!drawing!of!how!NexentaStor!provide!block!and!image!services!in!an! OpenStack!environment.!This!document!provide!detailed!steps!of!how!an!existing!NexentaStor!configuration! can!be!integrated!into!Mirantis!OpenStack.!! ! !!
Reference!Architecture!
! Both!Mirantis!OpenStack!and!NexentaStor!can!be!configured!to!provide!services!in!a!variety!of!ways.!To! ensure!that!the!best!possible!end!result!is!achieved!the!guidelines!and!best!practices!for!Mirantis!OpenStack! should!be!followed!to!configure!OpenStack.!! For!Best!Practices!for!NexentaStor!it!is!suggested!that!the!administrator!follow!the!guidelines!outlined!in!the! NexentaStor!User!Guide,!and!configure!a!dual/redundant!controller!setup!as!described!in!the!NexentaStor! High!Availability!Guide.! During!the!certification!and!functional!verification!of!NexentaStor!and!Mirantis!OpenStack!5.1!the!following! configuration!was!used:! • A!Mirantis!Fuel!Master!node! • One!OpenStack!Controller!node! • One!OpenStack!Cinder!node!! • 5!OpenStack!Compute!nodes! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Figure!1!Verification!setup!Node!configuration! When!NexentaStor!is!used!in!an!environment!that!is!managed!by!Mirantis!OpenStack!5.1!all!node! deployments!and!OS!configuration!must!be!performed!through!Mirantis!Fuel.!! NexentaStor!support!deployment!using!either!Centos!6.5!or!Ubuntu!12.04!LTS!when!using!OpenStack!Icehouse! based!releases.! Note:!This!document!assumes!CentOS!is!being!used!when!referencing!commandVline!utilities!and/or!OS!level! configuration!files!and!tools.! ! High!Availability!configurations! NexentaStor!support!Mirantis!OpenStack!running!in!either!High!Availability!mode,!or!in!‘nonVHA’!mode.!As! best!praxis!NexentaStor!itself!should!always!be!configured!in!high!availability!mode!with!two!redundant! controllers!as!documented!in!the!NexentaStor!HA!User!Guide.! ! Network!topologies! The!traditional!Nova!Networking!components!where!used!to!verify!the!solution!for!this!document.!However,! NexentaStor!will!work!with!both!Nova!Network!and!Neutron!modes!as!long!as!the!Compute,!Glance!and! Cinder!nodes!can!access!the!NexentaStor!server(s).! It!is!highly!recommended!that!the!storage!network!is!broken!out!as!either!a!physical!network,!or!as!a!separate! VLAN,!to!keep!the!traffic!segregated!from!other!traffic!types.! ! ! Figure!2!Network!used!for!functional!testing!
!
Installation!and!Configuration!
! When!NexentaStor!is!used!as!a!backend!storage!solution!for!OpenStack!the!guidelines!and!best!practices! published!on!by!Mirantis!apply.!The!deployment!of!Mirantis!OpenStack!should!be!done!through!FUEL,!and!the! deployment!should!pass!all!automated!health!checks.! ! After!installing!an!OpenStack!environment!using!Mirantis!Fuel!a!number!of!configuration!changes!!are! required!to!use!NexentaStor!as!backend!storage!for!Cinder!and!Glance:! • It!is!ideal!that!a!dedicated!Cinder!server!is!used!for!NexentaStor,!but!it!is!possible!to!configure! NexentaStor!as!a!part!of!a!multiVbackend!configuration!together!with!LVM!or!other!backends.!Please! see!the!OpenStack!Reference!Documentation!for!details!on!how!to!configure!multiVbackend!solutions.! • NexentaStor!must!be!configured!with!a!separate!NFS!share!for!Glance!Images! • If!the!NexentaStor!Cinder!NFS!drivers!is!used!a!separate!NFS!share!must!be!configured!for!Cinder! storage! • To!use!the!NexentaStor!Cinder!ISCSI!backend!it!is!suggested!that!a!separate!dataset/volume!is!created! to!keep!the!Cinder!volumes!separated!from!other!file!systems! • If!the!Cinder!is!configured!to!use!the!NexentaStor!NFS!backend!on!a!NexentaStor!4.0.x!appliance!a! separate!patch!must!be!applied!to!the!Cinder!driver!from!Upstream!Github!repositories.!This!is!to! correct!changes!in!behavior!between!NexentaStor!3.1.x!and!4.0.x!that!occurred!after!the!release!of! OpenStack!Icehouse! ! Software!requirements! • Mirantis!OpenStack!5.1!(OpenStack!Icehouse).!While!NexentaStor!is!supported!as!a!backend!store!for! Cinder!from!OpenStack!Essex!and!onwards!this!guide!focus!on!providing!details!based!on!OpenStack! Icehouse! • OpenStack!can!be!installed!on!any!Linux!distribution!supported!by!Mirantis!OpenStack!5.1.!! ! Prerequisites! This!guide!assumes!that!the!following!base!requirements!are!satisfied:! 1. NexentaStor!4.0.3!is!installed!and!configured!on!supported!hardware! 2. Mirantis!OpenStack!5.1!is!used!and!Mirantis!FUEL!is!used!to!deploy/manage!servers.!Technically!all! versions!since!OpenStack!Essex!are!supported,!but!this!document!is!specific!to!Mirantis!OpenStack!5.1! and!Icehouse! 3. The!environment!is!running!CentOS!7.x.!Again,!technically!any!Linux!version!that!support!Mirantis! OpenStack!5.1!or!OpenStack!Icehouse!are!supported,!but!the!documentation!is!specific!to!CentOS!7.x.! ! !Configure!OpenStack!Glance!to!store!Images!on!NexentaStor! The!OpenStack!image!repository!service!can!be!configured!to!store!images!on!NexentaStor!by!using!NFS! (Network!File!System).!The!Glance!Image!Cache!should!remain!local!on!the!Glance!server!for!best! performance.! ! Creating!a!new!share!in!NexentaStor!is!done!from!either!from!the!console!of!the!appliance!through!an! intelligent!command!line!shell,!or!by!the!Nexenta!Management!Console.! Below!is!a!screenshot!from!the!NexentaStor!Management!Console!that!detail!how!to!create!a!suitable!share! for!storing!Glance!Images.!As!an!example!we!are!creating!a!new!share!called!“glance_images”!in!the!“p1”! volume.!! ! ! ! ! ! ! ! ! ! ! ! ! ! Figure!4!Create!NFS!share!for!Glance!Image!Repository! ! ! ! ! ! ! ! ! ! ! ! ! ! Figure!3!Create!folder!to!store!Glance!Images!
! To!enable!the!Glance!server!to!access!the!new!folder!over!NFS!it!needs!to!be!shared.!This!is!done!through!the! folder!overview!in!the!NexentaStor!Management!Console!(see!below).! If!specific!permissions!are!needed!to!enable!the!Glance!user!to!access!the!Glance!Images!the!advanced! permissions!tab!can!be!reached!by!clicking!Edit!next!to!the!NFS!checkbox.! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Figure!5!Export!Glance!folder!through!NFS!
Configure!OS!to!mount!NexentaStor!file!system! To!configure!the!Glance!server!to!utilize!the!new!share!from!the!NexentaStor!appliance!it!must!be!mounted! on!the!Glance!server,!and!the!Glance!configuration!file!needs!to!be!updated!to!use!the!new!location!as!a! storage!location.! To!utilize!the!new!file!system!create!a!mount!point,!for!example!/var/lib/glanceVnexentastor!and!then!add!the! following!lines!to!/etc/fstab!on!the!Glance!server!to!ensure!it!is!mounted!at!boot:! Replace!“<NS>”!with!the!name!of!the!NexentaStor!appliance!and!“<VOL>”!with!the!name!of!the!Volume! where!you!created!the!glance_image!folder!above.!In!our!case!the!entry!would!look!like!this:! ! Finally,!confirm!that!the!NexenataStor!file!system!is!correctly!configured!by!manually!mounting!the!file!system! by!running!“mount!/var/lib/glanceVnexentastor”!on!the!Glance!server.!It!should!return!without!any!errors!and! the!file!system!should!now!be!mounted!under!/var/lib/glanceVnexentastor.! ! Note:!It!is!possible!that!the!CentOS!OS!images!does!not!have!the!required!packages!for!mounting!an!NFS!file! share!preVinstalled.!If!the!mount!command!fail!it!may!be!required!to!first!install!the!package!nfsVutils!by! running!“yum!install!nfsVutils”.!! ! Updating!the!Glance!configuration! In!the!Glance!configuration!file!/etc/glance/glanceVapi.conf!the!following!line!needs!to!be!changed:! ! ! so!that!it!points!to!the!new!image!repository!on!NexentaStor!! ! ! If!the!environment!already!had!Glance!images!stored!under!/var/lib/glance/images!these!can!now!be!copied! to!/var/lib/glanceVnexentastor/!.!It!is!suggested!that!a!copy!operation!is!used!to!allow!Glance!to!continue!to! function!during!the!operation.!! Restart!Glance!when!all!images!have!been!transferred!to!NexentaStor.!! ! ! ! ! After!the!restart!has!completed!Glance!will!now!utilize!NexentaStor!for!image!repository!and!the!old!images! can!be!removed.! !
# service openstack-glance-api restart
Stopping openstack-glance-api: [ OK ] Starting openstack-glance-api: [ OK ] # Mount Glance Image Repository from NexentaStor
<NS>:/volumes/<VOL>/glance_images /var/lib/glance-nexentastor nfs rw 0 0
# Mount Glance Image Repository from NexentaStor
nexentastor:/volumes/p1/glance_images /var/lib/glance-nexentastor nfs rw 0 0
filesystem_store_datadir=/var/lib/glance/images/
! NexentaStor!can!be!used!as!block!device!storage!for!OpenStack!Cinder!using!both!ISCSI!and!NFS.!When!ISCSI!is! used!the!Cinder!Volumes!are!created!as!ZVOLs,!block!devices,!on!the!NexentaStor!appliance.! As!best!practice!the!ZVOLs!should!be!stored!in!a!dataset/volume!that!is!unique!per!Cinder!backend!to!keep! them!separated!and!ensure!that!the!backend!drivers!do!not!interfere.! In!the!examples!below!we!are!using!the!dataset/volume!“p1”!to!store!the!ZVOLs!created!by!the!Cinder!ISCSI! driver.!! ! Configure!Cinder!to!use!the!NexentaStor!ISCSI!backend!driver! To!enable!the!Cinder!driver!for!NexentaStor!ISCSI!driver!the!following!additions!should!be!done!to!the!Cinder! configuration!file!/etc/cinder/cinder.conf:! ! ! ! ! ! ! Replace!the!HOSTNAME,!USER!and!PASSWORD!with!the!fully!qualified!hostname!or!IP!address,!and! administrative!credentials.!The!cinder.conf!file!should!have!appropriate!file!permissions!so!that!only!the! Cinder!driver!and!root!can!read!the!file!(If!unsure!run!“chmod!640!/etc/cinder/cinder.conf”).!The!value!for! nexenta_volume!should!be!set!to!the!name!of!the!dataset/volume!where!the!volumes!should!be!stored.! ! To!enable!the!NexentaStor!backend!configured!above!ensure!that!it!is!enabled!in!the!cinder.conf!file:! ! ! ! ! Note:!It!is!possible!that!the!CentOS!OS!image!does!not!have!the!required!packages!for!using!ISCSI!preV installed.!Please!ensure!that!the!iscsiVinitiatorVutils!package!is!installed!and!if!it!is!not,!install!it!using!“yum! install!iscsiVinitiatorVutils”! ! When!the!configuration!has!been!completed!Cinder!Volume!driver!needs!to!be!restarted!for!the!changes!to! take!effect:! ! ! ! After!the!restart!has!completed!Cinder!will!now!utilize!the!new!NexentaStor!ISCSI!backend!to!create!new! Cinder!Volumes!based!on!the!scheduling!and!weighting!rules.!! # /etc/init.d/opestack-cinder-volume restart Stopping openstack-cinder-volume: [ OK ] Starting openstack-cinder-volume: [ OK ] enabled_backends=nexentaiscsi-1 [nexentaiscsi-1] volume_driver=cinder.volume.drivers.nexenta.iscsi.NexentaISCSIDriver volume_backend_name=Nexenta_iSCSI nexenta_rest_port=8457 nexenta_host=<HOSTNAME> nexenta_user=<USER> nexenta_password=<PASSWORD> nexenta_volume=<VOLUME>
Configure!OpenStack!Cinder!to!use!NFS!from!NexentaStor!for!block!device!storage! NexentaStor!can!be!used!as!block!device!storage!for!OpenStack!Cinder!using!both!ISCSI!and!NFS.!When!NFS!is! used!the!Cinder!Volumes!are!created!as!regular!or!sparse!files!on!the!NexentaStor!appliance.! As!best!practice!these!files!should!be!stored!in!one!folder!per!Cinder!backend!to!keep!them!separated!and! ensure!that!the!backend!drivers!do!not!interfere.!If!the!configuration!utilize!multiple!Cinder!backends!these! must!all!have!separate!NFS!shares/folders.! Below!is!a!screenshot!from!the!NexentaStor!Management!Console!that!detail!how!to!create!a!suitable!share! for!storing!the!Cinder!Volumes!used!by!the!NFS!driver.!As!an!example!we!are!creating!a!new!share!called! “cinder_nfs”!in!the!“p1”!volume.!! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Figure!6!Creating!a!folder!for!storing!Cinder!Volumes!over!NFS!
! To!enable!the!Cinder!Volume!server!to!access!the!new!folder!over!NFS!it!needs!to!be!shared.!This!is!done! through!the!folder!overview!in!the!NexentaStor!Management!Console!(see!below).! If!specific!permissions!are!needed!to!enable!the!Cinder!user!to!access!the!Cinder!Volumes!the!advanced! permissions!tab!can!be!reached!by!clicking!Edit!next!to!the!NFS!checkbox.! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Figure!7!Share!folder!for!Cinder!Volume!storage!through!NFS!
Update!the!Cinder!driver!for!NexentaStor!4.0.x! OpenStack!Icehouse!was!released!before!NexentaStor!4.0.x!and!the!Nexenta!NFS!drivers!that!ship!with! Icehouse!are!not!compatible!with!NexentaStor!4.0.x.!The!upstream!drivers!in!GitHub!were!fixed!during!the! release!of!NexentaStor!4.0.1!and!this!will!not!be!a!problem!with!the!Juno!release.!However,!for!Icehouse!an! updated!NFS!driver!is!required.! To!download!and!install!the!updated!NFS!driver!run!the!following!commands!on!the!Cinder!Volume!server:! ! This!downloads!the!patched!NFS!driver!from!the!upstream!GitHub!repository!to!enable!Icehouse!to!work!with! NexentaStor!4.0.x.! ! ! Configure!Cinder!to!use!NexentaStor!NFS!Driver! To!enable!the!Cinder!driver!for!NexentaStor!NFS!the!following!additions!should!be!done!to!the!Cinder! configuration!file!/etc/cinder/cinder.conf:! ! To!enable!the!NexentaStor!backend!configured!above!ensure!that!it!is!enabled!in!the!cinder.conf!file:! ! ! ! ! ! [nexentanfs-1] volume_driver = cinder.volume.drivers.nexenta.nfs.NexentaNfsDriver volume_backend_name=Nexenta_NFS nexenta_shares_config=/etc/cinder/nexenta_shares enabled_backends=nexentanfs-1 # cd /usr/lib/python2.6/site-packages/cinder/volume/drivers/nexenta # mv nfs.py nfs.py.orig # wget \ http://raw.githubusercontent.com/openstack/cinder/46f3b9912af501aed5a2d91418311 0303f1c9cc5/cinder/volume/drivers/nexenta/nfs.py
! file!contains!all!the!information!the!Cinder!driver!requires!to!communicate!with!the!NexentaStor!appliance.! The!share!definition!file!has!the!following!format:! ! Replace!the!HOSTNAME,!USER!and!PASSWORD!with!the!fully!qualified!hostname!or!IP!address,!and! administrative!credentials.!The!share!definition!file!should!have!appropriate!file!permissions!so!that!only!the! Cinder!driver!and!root!can!read!the!file!(If!unsure!run!“chmod!640!/etc/cinder/nexenta_shares”).!!! ! Example!nexenta_share!!definition:! ! ! When!the!NexentaStor!driver!is!configured!the!Cinder!driver!will!automatically!mount!the!NFS!export!on!a!asV needed!basis.! ! Note:!It!is!possible!that!the!CentOS!OS!images!does!not!have!the!required!packages!for!mounting!an!NFS!file! share!preVinstalled.!If!the!mount!command!fail!it!may!be!required!to!first!install!the!package!nfsVutils!by! running!“yum!install!nfsVutils”.!! ! When!the!configuration!has!been!completed!Cinder!Volume!driver!needs!to!be!restarted!for!the!changes!to! take!effect:! ! ! ! After!the!restart!has!completed!Cinder!will!now!utilize!the!new!NexentaStor!NFS!backend!to!create!new! Cinder!Volumes!based!on!the!scheduling!and!weighting!rules.! ! ! # /etc/init.d/opestack-cinder-volume restart Stopping openstack-cinder-volume: [ OK ] Starting openstack-cinder-volume: [ OK ] nexentastor:/volumes/p1/cinder_nfs http://admin:nexenta@nexentastor:8457 <HOSTNAME>:/volumes/<VOLUME>/<FOLDER> http://<USER>:<PASS>@<HOSTNAME>:8457
!